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ABSTRACT 

The proposed quality assurance framework for higher education in Australia will, for the first time 
since the early 1990s, introduce a process of external quality monitoring through the auspices of the 
Australian Universities Quality Agency. There is little research that evaluates the impact of external 
monitoring on the experiences of staff, organisational and workplace culture, and perceptions of 
quality, within an institution. The paper will provide an evaluation framework for investigating the 
effects of external monitoring on elements of institutional culture and perceptions of quality in 
teaching and learning. The paper will explore issues of causality in evaluating academic audit as a 
quality management intervention in a tertiary institution, and across the sector as a whole. It will 
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explore criteria for evaluating the impact of audit on institutional culture and practices, as well as 
practitioners’ views of the impact of external quality monitoring on their own workplaces. 



PAPER PRESENTATION: AIR FORUM 2002 



When somebody’s watching: Researching the workplace impact of 
academic audit 

ABSTRACT 

The proposed quality assurance framework for higher education in Australia will, for the first time 
since the early 1 990s, introduce a process of external quality monitoring through the auspices of the 
Australian Universities Quality Agency. There is little research that evaluates the impact of external 
monitoring on the experiences of staff, organisational and workplace culture, and perceptions of 
quality, within an institution. The paper will provide an evaluation framework for investigating the 
effects of external monitoring on elements of institutional culture and perceptions of quality in 
teaching and learning. The paper will explore issues of causality in evaluating academic audit as a 
quality management intervention in a tertiary institution, and across the sector as a whole. It will 
explore criteria for evaluating the impact of audit on institutional culture and practices, as well as 
practitioners’ views of the impact of external quality monitoring on their own workplaces. 



4 

er|c 



Page 3 of 15 



Introduction 

Higher education around the world is facing unprecedented changes. These pressures are not new 
but the rate of change is exponential. Some of these changes have culminated in the explosion of 
quality assurance practices that preoccupy many of the worlds higher education sectors in OECD 
countries, hi the new global economy, accountability has taken centre stage, as we attempt to 
convince our global neiglibours of the merits of our public and private systems. 

Australia finds itself in a unique position. We are in an optimal pre-test phase, in the hiatus before 
an intervention is implemented, with the added convenience of the absence of any other 
intervention before it. This is a unique occurrence. Often in the evaluation of public policy we see 
the implementation of one policy overlapping the other. Causal factors of any subsequent change 
are difficult to isolate. However, this is not the case today in Australia as we embark on an external 
quality monitoring regime across the higher education system, for the first time this decade. 

Therefore the questions that arise are how will external quality audit change the system., public 
perception, our institutions, our practices and workplaces? What values underpin external quality 
monitoring and how are they currently aligned with values espoused in the higher education sector 
in Australia? What conditions are currently in place that will hinder or advance changes associated 
with this new intervention. This is a time for the Australian higher education sector to document 
preliminary perspectives that we will use in the future to judge how far we have come. 

When once the university was linked inextricably to conceptions of quality and integrity, 
universities today are faced with challenges of accounting for their activities to conform with 
external formulations of these terms. The university is no longer intrinsically enmeshed with the 
good intentions of its past. In addition, we face demands of multiple stakeholders who require 
accountabilities, efficiencies and cost benefits. We are required to be entrepreneurial, innovative, 
flexible, international, and yet these are not necessary contingencies to sway a sophisticated 
public’s perception of quality. How will the Australian higher education sector restructure its long 
standing edifices to meet these new challenges and what role will academic audit fulfil in our 
continuing metamorphoses? 

Thinking about audit involves discussions about the subjectivity of the watcher and the watched. If 
we agree that if we agree that the post-modem world is devoid of an objective eye, it is clear that 
the watcher will construct their perception while the watched simultaneously constructs the context 
for that perception. Perhaps we will declare “When somebody’s watching - everybody’s just 
dancing”. However those familiar with stage performance will know that the audience has a 
profound effect on the performer. The adrenaline felt by the performer as a consequence of the 
external gaze can drive further enhancement of what you do, and compel reflection on both the 
environment and the talents you will share with your audience. The subjective eye will enhance you 
if you are talented and denounce you if you are not - but perhaps only if the conditions are right. 
Mel Mark of Penn State University (Melvin, 2001) uses a model INUS in evaluation to describe the 
contingent or elliptical nature of causal knowledge. It describes the insufficient but necessary part 
of a causal package that itself is unnecessary but sufficient to cause a result. The underlying 
principle of the model is that the notion of cause has to be present with other things. This paper will 
extrapolate on some of these unnecessary but sufficient other things - that will determine the extent 
of change in our institutions and the sector as a result of academic audit. 



Background 

At the close of 1999, the Minister for Education, Training and Youth Affairs (DETYA) in Australia, 
Dr. Kemp announced the formation of an Australian Universities Quality Agency (AUQA) to 
establish a standard measure of accountability, accreditation, and quality assurance across all 
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sectors of higher education. The proposed quality assurance framework was to include provider 
comse accreditation, quality management of institutions, performance reporting and external audit 
processes (Kemp, Dec 10th, 1999). 

According to the minister, the quality assurance process for institutions that are self-accrediting will 
involve a whole of institution approach incorporating teaching and learning, research and 
management both onshore and offshore. The core activity will be an external audit based on a 
detailed self-assessment, including comparisons of standards. The role of the agency will include 
arranging periodic quality audits of, and providing reports on, self accrediting institutions and 
State/Territory accreditation authorities. Audits will be based on broad criteria, conducted against 
each institution's mission, therefore defining quality as ‘fitness for purpose”. (Kemp, 1999) 

This previous system was a reporting mechanism of quality assurance and improvement plans part 
of the annual submissions made to DETYA by universities. These plans typically outline the 
institutions goals, and strategies and indicators used to achieve and monitor these goals. Anecdotal 
evidence points to the inadequacy of this reporting system, in that the claims made in the reports are 
not monitored, or commented upon in any manner that directly affects the submitting institutions. In 
a recent report Anderson et al (2000) reiterated the weaknesses of these arrangements emphasising 
the lack of scrutiny, guarantee of standards and prospects for appeal for both graduates and the 
public. So we will no doubt go fi-om a toothless tiger, but towards what end? 

External Quality Monitoring: A model for academic quality assurance 

Monitoring in the guise of audits have in the main been perceived as a financial activity (Shore & 
Wright, 1999). These applications have been deployed more recently in areas of public policy such 
as the environment, health and education. They have functioned to provide assurance, 
accoimtability and management of risk (Power, 1 997). 

Types of external quality monitoring systems for higher education around the world include new 
hcence granting, accreditation, audits of quality assurance procedures, and periodic evaluation of 
viability (Harvey, 1998). According to Harvey (1998) there is a convergence of methodology 
involving external quality monitoring in most countries. In Australia this has translated in a slow 
projection of once honoured values bestowed on universities as “quality” institutions to an a priori 
value placed on external monitoring. As Penington, (1998) asserts, intrinsic quahties in truth and 
pure learning were “implicit in the ideals of scholarship [and] came to be seen by many in the 
academic community as a sufficient safeguard for the achievement of the quality of outcomes of 
investment by the community higher education” (p. 258). What has become evident in the rise of a 
“quality monitoring industry” in Australian post compulsory education is a discourse that parallels 
to that of ‘scholarship”, in themes of empowerment and self actualisation through quality 
management approaches. 

"One of the main claims made hy advocates of auditing is that it 'enables’ 
individuals and institutions to 'monitor ’ and 'enhance ’ their own perfonnance 
and quality, and to be judged by targets and standards that they set for 
themselves. This suggests that audit is an open, participatory and enabling 
process, is uncontentious and self evidently positive that there is not logical 
reason for objection ” (Shore & Wright, 1999), p. 560) 

Academic audit was not always the preferred choice for managing quality in the higher education 
sector in Australia. In 1998, David Pennington suggested models for the further assessment of 
quality including development of statements of purpose by individual institutions, increased 
benchmarking of “similar” institutions, increased use of student feedback and appraisal of research 
output. Market forces in the form of student choice of institutions for enrolment were seen as a 
major incentive for the development of quality systems in institutions, particularly with adjustment 
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of operating grants in accordance with major shortfalls in enrolments. The West report (Review of 
Higher Education Financing and Policy, 1998*) made clear “that there is no intention on the part of 
the current government to re-establish the former structure of surveillance of quality in the system” 
(p. 262, Penington, 1998). Obviously much can happen within one term of government and the 
passing of just a few years. 

According to the AUQA draft manual Version 0, it’s prime focus will entail quality audit for fitness 
for purpose. Audits will investigate the extent to which universities are achieving their mission and 
objectives, and the appropriateness of their quality assurance plans and processes. In support of 
these principles, for both agencies and institutions, the AUQA framework emphasises self-review 
(along the platforms of The Institution, Teaching and Learning, Research, Staffing, Resources, 
Results and Feedback) followed by external validation of the review results by AUQA. It also 
emphasises quality improvement as an intended outcome, alongside assurance, of the audit process. 



Importance of institutional culture for quality improvement 

During periods of change, it is important that we consider organisational culture and how the 
culture is being transformed from “old” to “new”. This transformation is apparent in moves towards 
a culture of enhancement and continuous improvement. This involves movement away from 
cultures of compliance to a standard, or what Shore and Wright (1999) refer to as the “docile 
auditable body” opposed to continuous improvement. Will external quality monitoring impact on 
changes evidenced in moving from one to the other, and will the impact be greater than would have 
occurred through a self assessment process anyway? 

The relationship of organisational culture to quality management practices is important because 
culture has been found to be a mediating variable between organisation systems and individual 
motivation. An organisational culture may be defined by the relatively enduring quality of an 
organisation’s internal environment which distinguishes it from other organisations (Katz, 1987). 
Work culture is a socially constructed ideology and practice with which workers state out a 
relatively autonomous sphere of action, created as workers confront the limitations and possibiUties 
of their jobs. It includes notions of a good days work and satisfying ad useful labour (Goode & 
Simon,’ 1987). 

Research has tested the fundamental assumption that organisational culture underpins and enables 
positive economic consequences for organisations such as increased employee commitment and 
cooperation, greater efficiency, improved job performance and better decision making (Langan-Fox 
& Tan, 1997). Doost, (1999) purports that value (i.e. quality) is comprised of staffs knowledge, 
positive interactions amongst themselves and with students, administrative support of staff and 
students, and public perception of how good a product the university is delivering. In their study on 
local governments Bowerman & Hawksworth, (1999) found that success of financial audit as a 
whole was dependent on collaborations between internal and external auditors. Lin, Madu, & Kuei, 
(1999) similarly found that characteristics of organisational culture are related to quality 
management practices. Organisational climate is a mediating variable between organisational 
systems and individual motivation and therefore directly influences behaviours that relate to 
productivity and retention, and attitudes, such as satisfaction and optimism Lin et al., 1999). 

The starting point for quality improvement requires organisational change and the identification of 
the need for change. This involves self knowledge of particular present, future and transition states, 
that range from, in relation to quality, a focus of compliance though to transformation and 
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improvement (Hill & Hazlett, 2001). Quality improvement requires a commitment to organisational 
learning as espoused by Barrow (1993) who argues that it should be the most compelling reason for 
undertaking quality initiatives and thus the learning agenda must be consciously incorporated into 
continuous improvement initiatives (cited in Hill, 2001). However, this needs to be tempered by 
setting operational infrastructure in place as a first priority. Hill’s case studies of organisations in 
transition demonstrated that an organisation that placed too much emphasis on cultural change and 
not enough on operational infrastructure and support systems failed to make the transition from a 
culture of compliance to one of continuous improvement. The author contends that this may be 
indicative of a lack of understanding on the part of management, again highlighting the importance 
of executive understanding in such transitions. 



Quality assurance and enhancement models that promote continuous improvement through 
evaluation have been used in the tertiary sector for some time, yet we are still unclear as to how 
these evaluations impact on orgamsational culture and decision making (Bunning, 1991) . There is 
still little research that asks why some analyses and types of information filter through to becoming 
part of policy cycles, and why organisations are receptive to particular forms of information and the 
way it is packaged, over others. Why do some organisations continue with strategies tha t are 
obviously not working, or fail to take actions which are clearly called for? Why do some 
organisations resist change so fiercely, defending the past and avoiding the future? 

Organisational culture is key to answering these questions. However, according to Langan-Fox & 
Tan, (1997), there is a dearth of studies that measure cultural change over time, or how a culture can 
be viewed in transition. This is probably due to the tendency of organisations to focus on immediate 
and current problems and a general lack of time and resources devoted to testing assumptions 
relating to impact of culture. Other difficulties involve assessing culture at differing levels at which 
it is manifested, the political issues involves, the existence of multiple subcultures, and the taken for 
granted nature of culture. 

In responding to change in higher education institutions may focus on system wide change while 
stopping short of focussing change conditions for individuals in the institutional workplace As 
Coaldrake & Stedman, (1999) state on the first page of their report. Academic Work in the twenty 
first Century: Changing Roles and Policies, 

"it is evident that universities more so than most organisations are built on a 
culture of individualism and academic personal autonomy, and so structural and 
policy changes must be considered in the light of patterns of academic and 
general staff work”, (p. 1). 

Studies by both Hill & Hazlett, (2001) and Langan-Fox & Tan, (1997) demonstrate that 
organisational culture is a critical factor in the success of quality management practices, and that in 
effect a new emerging culture is required for their successful implementation. In Langan-Fox & 
Tan's, (1997) study of a large financial services organisation, behaviours associated with the “old” 
culture included conducting business according to established procedures and historical practices, 
deferring to authority, not questioning norms and values, being rewarded for conformance. 
Behaviours typifying the ‘new’ culfore were characterized by a more customer-focused approach to 
business, achieving results via teamwork and staff involvement, questioning established practices 
and being rewarded for performance. These characteristics are paralleled in Bowden & Marton's, 
(1998) vision for the ‘university of learning’ that adapts Senge's, (1990) five dimensions of the 
learmng organisation to the university context. These comprise personal mastery as academics 
commitment to deep exploration within learned centred modes; systems thinking as academics 
develop a holistic view of core skills or competencies for learners; team learning as development in 
course and research teams; shared vision as a collective consciousness for what is common and 
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complementary; and mental models as inhibited communication in the spirit of free inquiry 
(Srikanthan, 200 1 ). 

External quality monitoring may affect underlying attitudes toward organisational learning by 
encouraging single loop learning (Bowerman & Hawksworth, 1999). Single loop learning is 
concerned with detection and correction of errors in the current operating system. Double loop 
learning involves the detection of errors with correction that involves finding solutions outside the 
current operating system (Argyris & Schon, 1974). Therefore double loop learning involves 
changes and improvements to the system itself. Deutero learning involves changing the ways of 
thinking about error detection and correction, so that what results is greater reflecting on the 
learning itself and improving learning capabilities (Romme & Witteloostuijn, 1999; Kay & 
Bawden, 1996). A continuous improvement culture requires double or deutero learning, however 
environmental conditions of trust and openness are crucial determinants of this kind of learning at 
both the individual and the organisational level (Carter & Palermo, 2000). At the very worst, 
organisational reactions to externa! quality monitoring may involve a fostering of culture that 
encourages coercion and compliance due to damaging effects on trust. Institutions may therefore be 
more interested in performing rather than learning and improving; performing for auditors may 
outweigh working towards real improvement and enhancement (Shore & Wright, 1999). 

The previous discussion therefore necessitates the question, will external quality monitoring in the 
guise of an academic quality audit produce: 

> a significant shift in values underpinning the institutional culture, identified by transitions 
firom cultures of compliance to cultures of improvement, or vica versa; 

> significant changes in staffs perceptions of “quality” in their institution? 

According to Stephen Kemmis, (1982) evaluation is a process of marshalling information and 
arguments which enable interested individuals and groups to participate in critical debate about a 
particular program. This assumes that staff will be interested and motivated to reflect on and 
evaluate their work. An evaluative work climate has to be developed if the evaluation is going to 
make a difference (Caulley, 1993). Therefore this calls for a focus on process to create a culture of 
ongoing monitoring, improvement and self evaluation. 

It is therefore cmcial that an evaluation firamework that assesses cultural transitions across 
compliance and improvement accompany a sector wide quality management intervention. Due to an 
emphasis on control and authority rather than improvement (Scott & Dinham, 2000), individuals 
may end up “talking the talk” as the audit process actively shapes behaviour (i.e. observation of a 
phenomena actively shapes that phenomena. Audit as it is currently conceived in the UK comes to 
shape the activities it is meant to merely oversee. Far firom being an observation of authentic work, 
the audit assessment becomes a formalised ritual, choreographed theoretical and staged (Shore and 
Wright, 2000). There is a danger that academic audit in Australia may therefore perpetuate the 
status quo, and hinder improvement, and at the very least create a climate of conservatism and 
caution, subsequently hindering innovation. If the purpose of higher education is to develop 
graduates who are critical and reflective, external quality monitoring should help not hinder that 
innovation (Harvey, 1998). Will quality audit affect change towards the directions required for 
higher education in the new knowledge economies? Will this change provide an environment that 
allows, as Ruth Dunkin (2001) asserts, learning spaces within and around universities where people 
can be creative, take risks and foster new ideas. 



Researching the Impacts of External Quality Monitoring on Workplace Culture 

There are commonly held views and ‘myths” about the effects of external quality monitoring that 
are more a reflection of the real fears and anxieties that individuals are confironted with during 
processes of change, than the result of evidence based evaluations of the process. Although it is 
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commonly claimed that external audit effectively depresses morale of staff in universities by 
introducing a sense of mistrust, (Maslen, 2000) there is little research that concludes this directly. 
Meaning and values differ from those academics have espoused traditionally and therefore this 
requires a shift in professional identity from “old independent scholar, inspiring teacher” to 
competitive performer. The increasing use of language and techniques associated with corporate 
management heralds a significant break from the principles of academic autonomy (Shore & 
Wright, 1999) . It dovetails an increasing preoccupation with “mode 2” as described by Gibbons, 
(1998) where there is an increased emphasis on teams over an individual approach, problem and 
issue based research that is transdisciplinary, and an increased connectivity in the work of 
universities, linking knowledge to more broadly based sources of knowledge. 

However the self regulatory approach should suit academics who have long espoused values of self 
government, autonomy and independence (Harvey 1998). External quality monitoring may provide 
a fi’amework within which self regulation is made standard across the sector. This may provide 
impetus towards driving change in the direction of forecasts by Coaldrake & Stedman, (1999) on 
the nature of changes to staff roles in universities They foresee, as their UK colleagues have, 
decreasing of distinctions between staff groups and a greater flexibility and diversity in engagement 
and expectations of academic staff These changes parallel changes in student demand and 
expectation. Students are more concerned with flexibility, ensuring the status of their qualification 
and access to high quality teaching and facilities (Coaldrake & Stedman, 1999) 

While maintaining the view that evaluation should be user driven, research by Palermo (2000) 
demonstrated that some prescription to external criteria is beneficial fi-om an individual perspective, 
in that they serve to focus activity and reduce anxiety in self assessment teams. Therefore what may 
be required is a balance between self assessment and external control. This may be realised in the 
acknowledgement that the locus of responsibility for quality assurance is the institution rather than 
the individual (Coaldrake & Stedman, 1999). It is the institutions that student look towards in a 
competitive market in choice making after all. 

The accountability in higher education in Australia push has been a light switch that has caused 
institutions to illuminate their concern for quality [Bogue , 1993 #73]. However the outcomes of 
this illumination are yet to be determined. In the UK there is little evidence that shows that quality 
monitoring is related to improved outcomes in teaching innovation (Harvey, 1998). There is no 
doubt that external quality monitoring gets people moving, but the question that is raised is to what? 
And to where? And to what end? 

Shore and Wright (1999), found intrinsic benefits of audit, as it provides a way of stating best 
practice and makes the implicit explicit (Jackson, 1997). At Monash University the effects of 
external quality monitoring were evident in more rigorous course approvals, increased awareness of 
learner centeredness and a shift in climate, (see Harvey, 1998). The latter outcome is the least 
tangible and yet the most important as it signals the significance of the intervention, and the 
probability that outcomes will continue to be influenced by that intervention. It signals an 
internalising of the principles of quality within the organisational climate or institutional psyche, 
and indeed within individual ideologies of staff and students. 



Importance Of Evaluating Effects 

According to Mitchell, (1996), there is a danger in swallowing rhetoric about the benefits of quality 
assurance having enhanced higher education (in America), solely due to the “current choms of 
voices” contesting it is so. He claims that much has been written about the “strong evidence” of the 
increased systematic scrutiny of teaching and learning and research, while being silent on exactly 
what this evidence is. This constant imposition of the theme of intrinsic improvement serves to 
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internalise a value system that is not made explicit and disarm dissent and critique. While not 
concurring with Wither’s (1995) conspiracy theory, the idea that quality assurance and monitoring 
is situated within a paradigm of thought that needs to be contextualised within political and social 
frameworks requires attention. As evaluators, we need to continuously retest our theories according 
to principled discovery and systematic analysis. External quality monitoring systems need to be 
evaluated in the same way any program or policy would be reviewed and evaluated for its merit, 
worth, contribution to improvement and knowledge development. 

Accountability, to government, international communities, is often the driver to the establishment of 
quality audit operations, as has been evidenced in Australia. While this is a noble endeavour, it is 
only part of a quality assurance process, as universities have responsibilities to other groups of 
stakeholders including their students. Learning requires an environment that allows for mistakes 
and is forgiving — universities should be, however governments are not. Responding in a market 
driven economy means that mistakes do matter. And so how does external quality monitoring 
impact on workplace behaviour is such an environm.ent? Shore and Wright (1999) offer a case study 
in Britain that relates significantly to a current controversy in Australia. They describe an individual 
lecturer who blows the whistle on the extent of falling standards, only to find themselves 
subjected to disciplinary proceedings. They describe the same pattern as evident in the health 
service with “gagging clauses” being incorporated into staff contracts. The logic of audits’ coercive 
accountability is that “no one dares to say that standards have declined because to admit this is 
tantamount to an admission of failure, and in a regime of competitive allocation of decline fimds, 
failure must be punished if “excellence’ is to be rewarded” (Shore and Wright) p. 668) 

Sample Evaluative Framework 

“not unlike the principle of the ‘black box’ flight recorder that is installed in all 
modem aircraft, if something goes amiss, the real time ’ data that it stores within 
itself can be used to find out what went wrong ’ 

(Kransdorff, 1996), p. 45) 

In order to explore some of these questions, I propose an evaluation fi'amework be conducted 
parallel to the audit cycle, that researches the impact of external monitoring on institutional 
workplaces. Research questions may include; 

Will external quality monitoring in the guise of an academic quality audit produce: 

^ a significant shift in values underpirming the institutional culture, related to continuous 
improvement and quality (as described by staff)? 

> a significant contribution to perceptions of an improvement culture in addition to outcomes 
achieved through self assessment processes in institutions? 

> Significant changes in staffs perceptions of “quality” in their institution? 

The following firamework is based on a pre-post test design where evaluations are conducted before 
and after an intervention. Where the design presented differs firom traditional pre-post test designs is 
in the additional evaluation phase proposed during the intervention. It also differs more importantly 
in acknowledging that the treatment ’ is not “controlled” by the researcher or evaluator. It adapts 
the experimental design by incorporating qualitative techniques of data collection via case studies 
and the important role of researcher as a participant observer. In this way the design is not 
positivist, nor purely qualitative. Instead the mixed method approach acknowledges the subjective 
attributes involved in research and evaluation while utilising a control group to observe behaviour 
in the absence of the intervention, thus increasing the reliability of any causal inferences made as a 
result of the evaluation. 
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Ideally an evaluative framework would comprise a case study that involves surveying staff at an 
institution of higher education using a pre-post test design . Additionally, in order to make causal 
inferences, staff at another institution would also be surveyed as a control group. The in-depth case 
studies will be complemented by qualitative data (collected at focus groups) from a wider group of 
higher education practitioners and senior academics in Australia, (see Table 1). 



Table 1 - Summary of proposed design: 

Time Stage INSTITUTION A: (case study) 



1 



1 



INSTITUTION 

(control) 



B- 



Focus groups Australia, assess sector — based perceived impact 

PRE-TEST . SURVEY STAFF PRE-TEST 

TEST (audit phase) Document analysis / 
interview panel members 



POST.. 

TEST 



pr;SURVEY STAFF POST-TESR^K^^ 



Focus group Australia, assess sector - based actual impact 



Stage 1 — Identifying criteria for evaluation, and assessing current climate: 

Self assessment criteria recommended by the various accrediting bodies tend to ignore the overall 
organisational effectiveness of institutions. This is a significant omission given that organisational 
effectiveness is a central concept in theory and practice (Pounder, 2000). Pounder (2000) employed 
the Competing Values Model of Organisational Effectiveness to produce a set of organisational 
effectiveness behaviourally anchored self rating scales in a study of Hong Kong higher education 
institutions. The Competing Values Model of Organisational Effectiveness comprises 8 factors; 
Adaptability - Readiness, Growth - Resource Acquisition, Productivity - Efficiency, Planning- 
Goal Setting, Stability Control, Information Management Communication, Cohesion - Moral and 
Human Resource Development. 

The behaviourally anchored scales can be used to measure the extent to which a stratified sample of 
staff in the organisation rate its effectiveness and readiness along dimensions that describe aspects 
of a conformance culture to a culture of continuous improvement. 

Pounder's, (2000) study showed that four scales in particular met stringent requirements of the valid 
and reliable self-rating of organisational effectiveness. These scales were: 

• Information Management - Communication; descriptive of mechanisms that have been 
introduced to enable the systematic dissemination of information; 

• Planning - Goal Setting; descriptive of coherent planning processes that allow cascading 
and tailoring at lower organisational levels; 

• Productivity - Efficiency; descriptive of behaviour concerned with the quantity or volume 
of production and operational costs; 

• Cohesion; descriptive of organisation’s steps towards creating positive cultural climate, 
including building staff morale, interpersonal relationships, teamwork and sense of 
belonging. 
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The criteria for assessment could be developed from adapting Pounder's, (2000) scales, however 
they have been presented here as one example of a tool that may be useful in measuring 
organisational culture in relation to quality improvement in a given organisation. There are many 
other sources and tools that could serve a similar purpose. For example, Langan-Fox & Tan, (1997) 
used a similar approach, however based their framework on a repertory grid technique borrowed 
from personal construct theory to measure peoples’ beliefs, values and assumptions within a given 
organisational context. This technique develops criteria or elements by asking staff to describe other 
staff in a systematic way, firstly by using different combinations of elements, and then contrasting 
these descriptions to form bipolar and descriptive constructs. In this way themes are established 
descriptive of an ‘old’ and ‘new’ culture, upon which scales of measurement may be based. 



A focus group comprising practitioners across the sector may also inform the pre-test condition in 
relation to identifying elements of the current climate (i.e. values and expectations), of practitioners 
in the sector. This initial investigation should explore individual’s perceptions about how audit has 
impacted on quality” in higher education, at differing levels, both individual and institutional / 
sectorial levels etc. 



Stage 2; Pre test - Survey quality current organisational climate 

Stage two involves interviewing all academic heads to assess current practices / initiatives in 
relation to quality of teaching and learning, and attitudes to different quality frameworks. A survey 
is developed based on the framework of variables informed by research findings in Stage 1. 
Perceptions of forecasts of impacts of the new system of audit are assessed. A randomly selected 
stratified sample of staff at both institutions surveyed in the pre-test phase. 

Stage 3: Test phase 

If possible, through the cooperation of the appropriate auditing agency, researchers gain access to 
documents submitted throughout the audit process, including minutes of meetings, deliberations etc 
for analysis. I also propose that it would be valuable to interview panel members after the audit 
process for their perceptions, and perceived impact from the process. Research outcomes in this 
stage would include identification of key decision making and evaluation processes during audit 
process, in order to evaluate the individual frameworks used by auditors, and identify perceptions of 
impact etc. 

Stage 4: Post test 

In order to ascertain impact of the audit intervention on a sector wide basis, a sector-wide based 
focus group is reconvened comprising quality managers and institutional researchers. Additionally, 
the staff in case study institution and control institution are re-surveyed using the same instrument 
developed in Stage 2. Post-tests occur after the audit process has been completed (including 
feedback). 



Concluding Remarks: overcoming limitations 

The most obvious limitations of the design presented are in the additional resources required to 
conduct a meta-evaluation of the quality audit process during the actual event, when often staffing 
and resources are already stretched to meet the demands of the actual audit process itself. However 
I would contend that as the design does not assume the evaluator is at arms length of the process or 
event, then the evaluation could be incorporated into the actual process, involving the same staff. In 
response to arguments about the dangers of meshing practitioner and researcher roles, I would argue 
that objectivity in research is a veneer in that the researcher is involved in subjective decision 
making and interpretation throughout the whole research process. This is evident from the 
formulation of the very questions and hypotheses asked, to the methodology chosen, to how results 
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are interpreted and generalised. Critical researchers have proven these arguments by espousing that 
truth makes reference to order, rules and values, and depends on logic and reasoning, all of which 
are unique to particular people or specific locales (Neuman, 1994). 

The quality manager who is coordinating the self assessment and portfolio production towards 
responding to an audit intervention, could also devise a limited amount of additional steps in the 
process that involve evaluative tasks as suggested in the framework. Data collection could be 
customised to include journal style entries, diary notes, vox pops and vignettes. 

The benefits of evaluating the academic audit by using a pre-post test design will be evidenced in 
benefits to the institutions, and to the sector as a whole. Assessing the impacts of quality audit is 
critical for informing policy on the development of an effective and efficient quality framework for 
the higher education sector in Australia. A key question raised by the current Minister of Education 
Employment and Training, Mr. B. Nelson’s paper. Higher Education at the Crossroads, is whether 
the existence of the Australian Universities Quality Agency (AUQA), as it is presently constituted, 
will be sufficient to guarantee quality of the Australian higher education system. The minister asks 
should the agency be given more teeth and do universities require more auditing, and from which 
Commonwealth and State authorities? Without evidence in relation to the impacts of quality audit 
on quality improvement in the sector as a whole, how can we reasonably contribute to the debate 
about the effectiveness and efficiencies the current operations for monitoring and improving quality 
in the higher education in Australia? And inversely, without this evidence, how can AUQA 
guarantee its effectiveness to the Australian public. 

Within an intuition learning is increasingly seen as a continuous work based activity, necessary to 
cope with changing demands from the external environment (Sambrook & Stewart, 2000) The 
evaluative framework offers opportunities to learning within the workplace environment, and wi thin 
the processes of ‘work’. Institutional learning arises from research, capacity building (personal and 
institutional) and practice (Senge, 1997). The development of learning capacities in organisations 
involves building genuinely shared visions (based on an understanding of shared values), fostering 
the ability of people to see larger patterns and understand interdependencies across systems, and 
increasing and developing capabilities of individuals to better enable them to become more aware of 
their own assumptions, i.e. develop critical thinking competencies. According to McAdam, Leitch, 
& Hamson, (1998), the learning organisation is characterised by the following five attributes: 

> It has a formal way of asking questions, seeking out theories, testing them and reflecting on them; 

^ It is properly, clear about its role, its future and its goals and is determined to reach them; 

y It is constantly reframing the world and its part in it; 

y Disappointment and mistakes are accommodated as part of the learning whole; 

^ It is a caring organisation, where everyone is encouraged to learn, and it makes that obvious. 

The learning university should adopt these principles in its approach to academic audit and it’s impact on 
organisational culture with a focus on building continuous improvement as an organisational capability. 
Without an evaluation framework paralleling academic audit, the sector and individual institutions alike have 
only hindsight to rely upon, when memories have faded and vested interests have grown. In the same way 
that AUQA will ask, “how do you know your are achieving what you have defined in your goals”, 
practitioners need to ask this of themselves in the context of the self review and audit process. And more 
importantly the sector, including staff, students and taxpayers, need to ask this of both the Commonwealth 
government and AUQA. 
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